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distance learning models and programs that may assist students who attend school from a remote location. 
To meet this need, this rapid evidence review sought to identify and report on what works in distance learning 
educational programming. After an extensive search and screening process, the What Works Clearinghouse 
(WWC) reviewed 36 studies spanning kindergarten through postsecondary education. Fifteen studies met the 
WWC Group Design Standards; of those, three met the Every Student Succeeds Act Tier 1 requirements. An 
analysis of where research has been conducted revealed that several distance learning programs for K-8 
students Met WWC Group Design Standards, but only one study of a distance learning program for high school 
students Met WWC Group Design Standards. In addition, a meta-analysis of studies with similar design 
characteristics (nine in total) found that, on average, students in the distance learning programs improved in 
the English language arts outcome domain but not in the mathematics domain. Although the results are 
promising, continued research using rigorous, randomized designs should be a priority. 


On March 11, 2020, the World Health Organization declared the spread of COVID-19 a global pandemic (World 
Health Organization, 2020), and the ongoing public health crisis has affected all aspects of daily life worldwide. 
Policies to mitigate the spread of the virus prompted school closures in all 50 United States and the four 
U.S. territories, disrupting the learning of approximately 50.8 million public school students in the 2019/20 
academic year (Education Week, 2020a). The 2020/21 academic year is now underway, and 74 of the 100 largest 
school districts started the school year fully remote, affecting more than 9 million students (Education Week, 
2020b). 


To continue to educate American students, educators and school administrators are working to understand the 
variety of available distance learning models and programs, defined broadly to include any digital educational 
material designed to be accessed outside the traditional in-person school setting. Existing distance learning 
programs encompass a wide range of models: fully online schools, individual courses offered to high school or 
postsecondary students entirely online, hybrid courses that offer a mix of in-person and remote instruction, and 
programs that supplement existing curricula with online instruction or offer additional online practice for a specific 
academic area (Digital Learning Collaborative, 2020). 


In order to quickly and adeptly make use of the variety of available digital programming, educators and 
administrators need resources to help understand the effectiveness of these programs. They also need 


1 This is a revised version of the original report released in January 2021. For the Math Snacks program, the effect size, statistical 
significance, and Every Student Succeeds Act Tier rating has been revised. For the Education Program for Gifted Youth, the WWC rating 
was revised, and as a result, one additional effect size was added. The changes to these two programs resulted in small changes to the 
meta-analytic findings and moderator results. Additionally, the following program profiles have been modified to reflect small 
inconsistencies: Intelligent Tutoring for the Structure Strategy, MindPlay Virtual Reading Coach, and Reading Plus. The evidence gap map 
titles were updated to include the number of findings, rather than studies, presented in the maps. Finally, small copyedits and 
grammatical revisions were made throughout. None of the revisions resulted in substantive changes to the conclusions. 
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information about programming characteristics to ensure that programs are suitable for specific settings. 
Educators may wonder if a program offers a full curriculum or is intended to be supplemental. Does it require 
synchronous or asynchronous interaction with a teacher, or is it fully automated? What kind of personalization or 
adaptive capabilities does the program have? Unfortunately, educators and administrators lack information about 
the critical characteristics of distance learning programs, their quality and effectiveness, and the supporting 
evidence (Tosh, Doan, Woo, & Henry, 2020). 


With the goals of better understanding what works in distance learning and providing relevant information to 
educators and administrators, the Institute of Education Sciences (IES) put out a call for studies that evaluated the 
effects of distance education practices and programs on student academic outcomes (National Center for 
Education Evaluation and Regional Assistance, 2020). The What Works Clearinghouse (WWC) teams worked with 
the Education Resources Information Center (ERIC), IES’s digital library of education research, to screen and 
prioritize studies for WWC Group Design Standards review (WWC, 2020a). Citations and abstracts were screened 
in accordance with standard WWC eligibility criteria, as well as supplemental criteria identified specifically for this 
review (these supplemental criteria are described in box 2 and the technical appendix). Of the 932 studies found 
in the initial search, 266 were retained for full-text screening (see box 2 for a full description of the screening 
process and criteria). Because a primary goal of this review was to provide practitioners with rigorous evidence 
on effective distance learning programs as quickly as possible, only randomized controlled trials (RCTs) of 
programs with at least one positive and statistically significant finding were retained following the screening 
(definitions of key terms, such as RCT, are provided in box 1). More information on the search and screen process, 
as well as the number of studies screened out due to various criteria, is provided in figure A—1 in the technical 
appendix. Thirty-six studies were identified for review.’ Five of these studies had been previously reviewed by the 
WWC; the remaining studies were reviewed by WWC contractors using the Review of Individual Studies Protocol 
version 4.0 (RISP v4.0), and underwent a thorough peer review. This report summarizes the results of the WWC 
review process. 


Throughout the process, the WWC was guided by the four research questions listed below. This report was 
structured around the answers to those questions. The report ends with a characterization of the state of the 
distance learning field, based on the studies reviewed, and some potential future steps. 


Research questions 

1. How many distance learning programs have been evaluated in studies that Met WWC Group Design 
Standards? What are the characteristics of these studies (such as design, sample, and outcomes) and the 
programs’ components (such as program type, content coverage, and teacher interaction style)? 

2. Where is evidence needed to address gaps in the distance learning literature? 

3. What is the overall average effect of distance learning programs that meet WWC standards on English 
language arts (ELA) and mathematics achievement? 

4. Do any of the programs’ components explain differences in effectiveness? 


The search and screening procedure originally identified 35 citations for review by the WWC. WWC contractors could not identify full 
text for one of the citations and substituted two recently released publications from the same study team for the citation. Separately, 
during the review process, the WWC determined that there were two instances of multiple publications describing the same study; in 
these instances, citations were combined for a single review. Two additional publications described multiple treatment conditions, so 
separate reviews were conducted for each treatment comparison contrast within those publications. This resulted in a total of 36 WWC 
reviews. The full list of citations with final WWC dispositions is provided in table A—3 of the technical appendix. 
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Box 1. Key terms 


Business-as-usual comparison group. A group in which students receive the same instruction as would be expected in the 
absence of the intervention. 

Distance learning. Digital educational material designed to be accessed outside the traditional, in-person school setting. 
Effect size. The extent of an intervention’s impact on students who receive distance learning relative to the comparison 
group. 

ESSA Tier 1. A measure of the effectiveness of an intervention based on the Every Student Succeeds Act definition. To achieve 
Tier 1 status, a study must have at least one finding that (a) Meets WWC Group Design Standards Without Reservations, (b) 
has a sample of at least 350 students from multiple schools, and (c) is statistically significant and positive. 

Evidence gap map. A visualization method that depicts the number of studies on a matrix with thematically important 
dimensions represented on each axis and by study markers. An evidence gap map is a concise and user-friendly figure that 
illustrates the types of studies that have and have not been conducted. 

Meets WWC Group Design Standards Without Reservations. A descriptor that applies to a study that met the highest quality 
design standards, such as a randomized controlled trial (RCT) with low attrition. 

Meets WWC Group Design Standards With Reservations. A descriptor that applies to a study that has high-quality design 
standards, such as an RCT with high attrition or a quasi-experimental design, both with baseline equivalence between the 
intervention and comparison conditions. 

Meta-analysis. The process and statistical technique of combining effect sizes from multiple independent studies. 
Randomized controlled trial (RCT) design. A design that assigns students participating in a study to either an intervention or 
comparison group condition using a random process. 

WWC Group Design Standards. Methodological guidelines that dictate how the WWC determines the quality of an 
evaluation. 


Findings 

The search and screening procedure identified 36 studies that were potentially eligible for WWC Group Design 
Standards review (figure 1). The WWC’s review process provides a rating of a study’s methodological quality. The 
highest methodological quality rating a study can receive is Meets WWC Group Design Standards Without 
Reservations; this rating is reserved for studies that meet the highest threshold for quality and conveys the most 
confidence that the intervention caused changes in the intervention group’s academic performance. The second- 
highest methodological quality rating a study can receive is Meets WWC Group Design Standards With 
Reservations; this rating indicates that the intervention most likely contributed to changes in academic 
performance, but some caution may be required. Studies that Wet WWC Group Design Standards With or Without 
Reservations are featured in the program profiles, evidence gap map, and meta-analysis. Studies that were 
reviewed by the WWC but fall short of quality standards received a rating of Does Not Meet WWC Group Design 
Standards and are included in the evidence gap map designed to illustrate where research has previously been 
conducted. Finally, studies that did not meet basic design criteria—for example, not measuring a relevant 
outcome or lacking a comparison group—are not eligible to be reviewed by the WWC. 


Of the 36 studies that were identified as potentially eligible for WWC review, 12 Met WWC Group Design 
Standards Without Reservations, three Met WWC Group Design Standards With Reservations, nine received a 
rating of Does Not Meet WWC Group Design Standards, and 12 were not eligible for WWC review. The full 
methodology is described in the technical appendix. 
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Note: Initially, 35 citations were identified for WWC review. Due to difficulties retrieving full text, the combination of citations describing a single study, and 
studies including multiple treatment conditions, the total number of citations included in this review is 36. The full list of citations with final WWC dispositions 
is provided in table A—3 of the technical appendix. 


Of the 15 studies that Met WWC Group Design Standards With or Without Reservations, 11 were conducted with 
K-8 students, one with high school students, and the remaining three with postsecondary students. Four 
programs were free of cost, and eight programs had associated costs (programs ranged substantially in their 
estimated costs, from $15 per student to several thousand dollars per student). For the remaining three programs, 
the WWC could not identify any information about program cost in publications or through an Internet search. 
Eight of the 15 studies measured outcomes in the ELA domain, four measured outcomes in the mathematics 
domain, and the remaining studies measured general achievement or progress at the postsecondary level. Nine 
of the 15 studies used a business-as-usual comparison group in which students received traditional, in-person 
instruction. In the other six studies, treatment students received an enhanced or optimized version of a distance 
learning program, while students in the comparison condition received a base version of the program. A list of the 
studies’ characteristics can be found in table A—1 of the technical appendix. 


Box 2. Data sources, sample, and methods 


Data sources. Studies that Met WWC Group Design Standards With and Without Reservations contributed to the program 
profiles, evidence gap map, and meta-analysis. Studies that did not meet WWC Group Design Standards contributed only to 
the evidence gap map. 


Sample. A total of 36 studies were identified for WWC review; 24 of these studies were eligible for WWC review, 15 of which 
Met WWC Group Design Standards With or Without Reservations. 


Methodology. The WWC conducted a systematic review using a rapid evidence review technique, relying on two methods to 
identify potential studies. The first method was a call to education researchers and stakeholders to submit studies that 
evaluated distance learning programming (National Center for Education Evaluation and Regional Assistance, 2020). The 
second method was a traditional search of the ERIC database using key terms. These methods yielded 932 citations, which 
are available on the WWC’s Studies of Distance Learning webpage. Titles and abstracts for all citations were screened for the 
following: a sample of students in grades K—16, a location in the United States, publication in 2010 or after, a measure of a 
student outcome, and availability in the ERIC database. After this first screening, 266 citations of the 932 studies found in the 
initial search were retained and attempts were made to retrieve the article PDFs. The studies then underwent two rounds of 


full-text screening. The first round sought to identify studies using criteria specifically designed to facilitate a rapid evidence 
review process: (a) the study had been peer reviewed, (b) the study used an RCT design, (c) the study reported at least one 
positive and statistically significant finding, and (d) programs were fully online.* Next a second, full-text eligibility screening 
was conducted using the Review of Individual Studies Protocol (RISP), version 4.0. This yielded 36 RCTs that were potentially 
eligible for WWC review (see figure 1). All 36 studies then underwent the usual WWC review process. Of the 36, 24 were 


3 Note that eligibility criteria required studies to have at least one significant positive student finding, but the eligibility of the outcome 
was not assessed at the screening stage. As a result, some studies that received full reviews did not have any eligible significant positive 
findings. Similarly, studies were required to examine fully online programming. After eligible studies received a full review, it was 
determined that several of the programs included in this review did include in-person instruction in addition to remote instruction. 
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eligible and reviewed using WWC Group Design Standards, version 2.1 or later.* In addition to the typical review process and 
effect size estimation, the WWC also coded components of the programs. 


Analysis. Studies that were eligible for review were used to create an evidence gap map. Studies that Met WWC Group Design 
Standards were synthesized using a fixed effects, meta-analytic model, weighting the effect sizes by the inverse of the effect 
size variance (WWC, 2020b). Moderator analyses of the program components were also conducted. 


The WWC also extracted information on key components (box 3) of each distance learning program. The 
components were determined through a discussion with content experts and included whether the program 
provided a complete course or was supplemental, whether it covered broad or narrow content, the type of 
interaction (teacher-led synchronous, teacher-led asynchronous, or automated/not teacher directed), whether 
the program was personalized or differentiated based on student performance (adaptive or nonadaptive), and 
whether the program was gamified (was interactive, had goals, and followed a story arc). The distribution of these 
components across the studies that Met WWC Group Design Standards is described in figure 2. Most of the 
programs included in the review were supplemental (87 percent), covered a broad content area (67 percent), and 
were not gamified (93 percent). A slight majority had no teacher-directed interaction (53 percent) and were 
adaptive (53 percent). A full description of the program components, how they were operationalized, and 
additional information can be found in table A—2 of the technical appendix. 


Box 3. Description of program components 


Program type. Full curricula cover an entire course or a semester’s worth of content. Supplemental curriculum programs are 
used in conjunction with a full curriculum. 


Content coverage. Programming with narrow content coverage is limited to a few substantive topics or focuses on a specific 
skill or strategy. Programming with broad content coverage includes a range of topics or skills. 


Teacher-student interaction. Programs with teacher-led synchronous instruction include an educator who is physically 
distanced from students but is engaging in real-time, live instruction. Teacher-led asynchronous instruction may occur via 
recorded interactions or through an interactive program that does not require the instructor and students to be engaged at 
the same time. Other programs may be fully automated, may follow a preprogrammed set of activities or modules, and may 
not require interaction from any live instructor. 


Personalization. Adaptive programs are personalized and specific to the performance of the participant. Adaptive 
programming can be fully automated by a program or implemented by a teacher. 


Gamification. Gamified programs are interactive, include specific goals, and often include a story arc. 


4 Two studies that were previously reviewed prior to the release of version 4.0 of the WWC Group Design Standards: Roschelle et al. 
(2016) was reviewed under version 3.0, and Heppen et al. (2011) was reviewed under version 2.1. 
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Figure 2. Program component characteristics of studies that Met WWC Group Design Standards 
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Note: Program components are at the study level. 


Profiles of distance learning programs that met ESSA Tier 1 rating requirements 

The following section presents profiles of each of the three programs with studies that Met WWC Group Design 
Standards Without Reservations and ESSA Tier 1 rating requirements. The profiles include a brief program 
description, program components, program cost, duration, and relevant information for any positive, statistically 
significant findings, including the WWC’s improvement index. Profiles of the remaining programs that Met WWC 
Group Design Standards but did not meet ESSA Tier 1 rating requirements can be found in the technical appendix. 
Please consult the WWC’s Review of Individual Studies database for full study review details. 
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ASSISTMENTS 


PROGRAM DESCRIPTION PROGRAM FEATURES 

PROGRAM TYPE DELIVERY METHOD 
ASSISTments is an online platform for mathematics homework and y | 9 Is 
related teacher training. Students use the platform to complete their OO : 
mathematics homework and are provided with immediate feedback is Oo : ‘ am) 
via the platform while they solve assigned problem sets. The Supplemental Hybrid 


platform provides teachers with reports on student performance 
and commonly missed questions. For this study, the ASSISTments 


software was loaded with all homework problems from all textbooks © TEACHER-STUDENT INTERACTION COST 
in use among intervention schools, as well as mathematics extension | 10 
activities called “skill builders,” covering more than 300 topics 
appropriate for grade 7. QoQ 

ue ray | 

Teacher-led synchronous Free 

e Content Coverage: Broad e Gamification: No Duration: Schools implemented ASSISTments for 
¢ Participant Grade Level: Grade 7 e Adaptive: Yes two academic years. Student cohorts used the 


program for one academic year each. 
FINDINGS | This study Meets WWC Group Design Standards Without Reservations and has at least one statistically significant 
positive finding. 


Outcome measure Comparison Sample Significant? Improvement index 
General mathematics achievement | ASSISTments versus business as usual 2,728 students Yes "ei 
-50 +50 
CITATION 


Roschelle, J., Feng, M., Murphy, R. F., & Mason, C. A. (2016). Online mathematics homework increases student achievement. AERA 
Open, 2(4), 1-12. https://eric.ed.gov/?id=ED575159 
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INTELLIGENT TUTORING FOR THE 


STRUCTURE STRATEGY 


PROGRAM DESCRIPTION PROGRAM FEATURES 

Intelligent Tutoring for the Structure Strategy (ITSS) is a supplemental, 

online program for students in grades K—8 to develop literacy skills and PROGRAM TYPE DELIVERY METHOD 

understand factual texts used in everyday settings. Students learn 

how to use text structure, key words, and logical structure to better Gg is’) 

understand and recall information. In particular, ITSS highlights five —_— ‘am 

main text structures that (a) make comparisons, (b) present problems SUBAIE EHR Fully pale 
; A pp ly 

and solutions, (c) link causes and effects, (d) present sequences, and 

(e) describe things, people, creatures, places, and events. The WWC 

intervention report on ITSS can be retrieved from 

https://ies.ed.gov/ncee/wwc/InterventionReport/703. TEACHER-STUDENT INTERACTION COST 


The WWC reviews on ITSS can be retrieved from 
https://ies.ed.gov/ncee/wwc/Study/77453, {ZG 


https://ies.ed.gov/ncee/wwc/Study/86126 aa = 
https://ies.ed.gov/ncee/wwc/study/86233, Automated/Not teacher directed Some cost 
and https://ies.ed.gov/ncee/wwc/study/89695. 
e Content Coverage: Narrow © Gamification: Duration: Students use the program throughout the academic year, for 
e Participant Grade Level: No 30-45 minutes per session, one to three times a week. 

Grades 4, 5, and 7 e Adaptive: Yes 


FINDINGS | The following studies Meet WWC Group Design Standards Without Reservations and have at least one statistically 
significant positive finding. 


Outcome measure Comparison Sample Significant? Improvement index 
Comprehension ITSS versus 4,196 students Yes Ns 
business as usual 50 +50 
CITATIONS 


Wijekumar, K. K., Meyer, B. J. F., & Lei, P. (2012). Large-scale randomized controlled trial with 4th graders using intelligent tutoring of 
the structure strategy to improve nonfiction reading comprehension. Educational Technology Research and Development, 60(6), 987— 
1013. https://eric.ed.gov/?id=EJ986753 

Wijekumar, K., Meyer, B. J., Lei, P. W., Lin, Y. C., Johnson, L. A., Spielvogel, J. A., Shurmatz, K. M., Ray, M., & Cook, M. (2014). Multisite 
randomized controlled trial examining intelligent tutoring of structure strategy for fifth-grade readers. Journal of Research on 
Educational Effectiveness, 7(4), 331-357. https://eric.ed.gov/?id=EJ1041354 

ADDITIONAL CITATIONS | The following studies Meet WWC Group Design Standards With Reservations or did not have at least 
one Statistically significant positive finding. 

Meyer, B. J., Wijekumar, K., Middlemiss, W., Higley, K., Lei, P. W., Meier, C., & Spielvogel, J. (2010). Web-based tutoring of the 
structure strategy with or without elaborated feedback or choice for fifth- and seventh-grade readers. Reading Research 

Quarterly, 45(1), 62-92. https://eric.ed.gov/?id=EJ871741 

Meyer, B. J. F., Wijekumar, K. K., & Lin, Y. (2011). Individualizing a web-based structure strategy intervention for fifth graders’ 
comprehension of nonfiction. Journal of Educational Psychology, 103(1), 140-168. https://eric.ed.gov/?id=ej914858 

Wijekumar, K., Meyer, B. J. F., & Lei, P. (2017). Web-based text structure strategy instruction improves seventh graders’ content area 
reading comprehension. Journal of Educational Psychology, 109(6), 741-760. https://eric.ed.gov/?id=EJ1149967 
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ONLINE ALGEBRA | 


PROGRAM DESCRIPTION PROGRAM FEATURES 
PROGRAM TYPE DELIVERY METHOD 

The online algebra | curriculum uses electronic textbooks with interactive 
features such as computerized direct instruction and practice problem 
sets with automated feedback. Students are provided a variety of = 
multimedia materials and a discussion board and communicate with ‘ ; 

5 ° : Full curriculum Fully online 
teachers using a messaging feature. Online teachers grade homework, 
monitor student understanding, promote student engagement, conduct 
online discussions, and monitor student performance. Students use the TEACHER-STUDENT INTERACTION Cost 
software in schools, where an on-site proctor supports students. 
Implementation data showed that online teachers spent less time 
working directly with students and that on-site proctors spent more time 
communicating with students about mathematics than planned. Cc 

Teacher-led asynchronous Some cost 

e Content Coverage: Broad e Gamification: No Duration: Students used the curriculum 
¢ Participant Grade Level: Grade 8 e Adaptive: No for one full academic year. 


FINDINGS | This study Meets WWC Group Design Standards Without Reservations and has at least one statistically significant 
positive finding. 


Outcome measure Comparison Sample Significant? Improvement index 
Advanced course-taking Online algebra | versus 440 students Yes oa 
business as usual 50 +50 
General mathematics Online algebra | versus 440 students Yes gu 
achievement business as usual 50 750 
CITATION 


Heppen, J. B., Walters, K., Clements, M., Faria, A. M., Tobey, C., Sorensen, N., & Culp, K. (2011). Access to algebra |: The effects of 
online mathematics for grade 8 students [NCEE 2012-402]. National Center for Education Evaluation and Regional Assistance. 
https://eric.ed.gov/?id=ED527394 


Findings from evidence gap maps of studies eligible for WWC review 


To understand where more evidence is needed—and to address gaps in studies of distance learning programs— 
the WWC created an evidence gap map of all 24 studies eligible for WWC review (figure 3) and a separate gap 
map of the 15 studies that Met WWC Group Design Standards (figure 4). One postsecondary study included 
findings in two domains, both of which are included in the gap maps. These maps allow users to separately 
understand what types of distance learning programs and outcomes were studied (figure 3), as well as what types 
of programs have an existing evidence base (figure 4). The evidence gap map of all studies eligible for WWC review 
is particularly informative for researchers as it indicates what programs require additional rigorous study. The 
evidence gap map of studies that Met WWC Group Design Standards is especially informative for administrators 
and practitioners who may be looking for studies of effective programs that meet their specific needs and context. 


It is important to note that only RCTs with at least one positive, statistically significant finding were deemed eligible 
for a WWC review; unlike in other WWC reviews, studies with a quasi-experimental design or RCTs without 
positive findings were not eligible. As a result, this gap map clarifies gaps among RCTs of effective distance learning 
programs rather than gaps in the distance learning literature at large. 
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Box 4. Interpreting evidence gap map cells 


The top left cell in the evidence gap map of eligible studies in figure 3 shows the number of eligible studies that evaluate 
distance learning programs offering full curricula or supplemental curricula and measure English language arts (ELA) 
outcomes. The small red square in that cell conveys that only one study of a supplemental program that measured ELA 
outcomes for high school students was eligible for WWC review. The blue circle indicates that eight studies of supplemental 
programs that measured ELA outcomes for K—8 students were eligible for WWC review. No RCTs of full curricula that 
measured ELA outcomes at any grade level were eligible for WWC review. The same cell in the evidence gap map in figure 4 
shows the number of studies that Met WWC Group Design Standards, evaluated full curricula or supplemental curricula, and 
measured ELA outcomes. Comparing this cell across the two gap maps, we see that of the eight studies of supplemental 
distance learning programs for K-8 students measuring ELA outcomes, seven of them Met WWC Group Design Standards. 


Several characteristics were plotted in the evidence gap maps. The rows represent the six outcome domains 
examined in these studies. The columns of the evidence gap map represent program components of the 
interventions studied, such as whether the program provided a full curriculum or is a supplemental program. The 
levels of each component are mutually exclusive—that is, a program cannot fall into more than one component 
category. The shape of the marker corresponds to the grade level, and the size of the marker and the numeral 
inside it correspond to the number of studies. An example of how to interpret each cell is provided in box 4. The 
map revealed several interesting findings, which we describe below by dimension. 


Figure 3. Evidence gap map of distance learning RCTs eligible for WWC review (n = 25) 
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Grade level. The majority of the eligible studies examined programming for K-8 students (68 percent), and more 
than half of those (11 of 17) Met WWC Group Design Standards. The distance learning programs for students in 
grades K-8 are diverse in terms of program components, and there are at least two studies in each component 
level. Of the K-8 programs that Met WWC Group Design Standards, only one provided a full curriculum. 


One of the most pronounced gaps in the map arises from the fact that there are only two eligible studies of 
programs for students in grades 9-12, and only one Met WWC Group Design Standards. These two were the only 
studies identified for WWC review during the screening stage. The programs were both supplemental programs— 
one with broad academic content coverage and one with more narrow coverage. Neither program was adaptive 
or gamified. 


Five postsecondary studies, with six findings, were eligible for WWC review, three of which Met WWC Group 
Design Standards. One study examined outcomes in both general academic achievement and progress in school. 
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More studies were identified for WWC review (n = 15), but many of these studies did not include an eligible 
outcome.” Four of the five programs were supplemental distance learning programs covering broad academic 
content. One was a teacher-led asynchronous program, and the others were not teacher directed. None of the 
programs were adaptive or gamified. 


Figure 4. Evidence gap map of distance learning RCTs that Met WWC Group Design Standards (n = 16) 
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Outcome domain. The majority of studies eligible for WWC review examined ELA and mathematics outcomes 
(68 percent). Few studies examined science, social-emotional outcomes, general academic achievement, or 
progress in school, and those that examined science and social-emotional outcomes Do Not Meet WWC Group 
Design Standards. The studies that measured ELA and mathematics outcomes were primarily studies of programs 
for students in grades K—-8 (with the exception of one program for high school students). On the other hand, 
general academic achievement and progress in school were assessed only by studies of postsecondary 
programming. 


The lack of studies that examine social-emotional outcomes is especially notable, as one concern with distance 
learning is the social-emotional health of the students (Education Week, 2020c; Walker, 2020). Students who are 
taught remotely may not be able to engage with the wider community, and teachers and staff are not able to 
check in on the social-emotional state of their most vulnerable students. 


Distance learning program components. An examination of the components of distance learning programs shows 
that 16 percent (four of 25) of the evaluated programs provided a full curriculum rather than supplemental 
programming. All programs that Met WWC Group Design Standards and used teacher-led synchronous instruction 
were for students in grades K—-8 and high school. The postsecondary programs used asynchronous instruction or 
had no teacher-led instruction at all. All of the adaptive programs, as well as the three gamified programs—only 
one of which Met WWC Group Design Standards—were for K—8 students. It is unclear whether programs for high 
school and postsecondary students do not use these innovative approaches of gamification and personalization 
or if such programs have not been evaluated by RCTs or have not been found to have positive, statistically 
significant effects. 


5 Several studies of postsecondary distance learning measured outcomes that do not meet the requirement, outlined in RISP v4.0, that 
academic achievement measures must be final course grades, final or midterm examination scores, or grade point averages. Academic 
achievement assessed via tests, quizzes, or non-final examinations that do not fall in any of the other academic domains is not eligible for 
review. 
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Findings from meta-analyses of studies that Met WWC Group Design Standards 


To answer the third research question—what is the overall average effect of distance learning programs that Met 
WWC Group Design Standards—the WWC sought to summarize studies using the technique of meta-analysis. A 
total of nine studies contributed to the meta-analysis: five with outcomes in the ELA domain and four with 
outcomes in the mathematics domain. A full description of the meta-analysis methods is provided in the technical 
appendix. 


The results of the meta-analyses are presented in table 1. For ELA, five studies—which together included 
6,853 students—Met WWC Group Design Standards. The average effect size for ELA was 0.20 (SE = 0.03) and was 
statistically significant (p < .05). This effect size translates to an improvement index of +8, meaning the average 
student participating in one of the eligible distance learning programs is predicted to score eight percentile ranks 
higher in ELA than the average student receiving traditional, in-person instruction. For mathematics, four 
studies—which together included 5,393 students—Met WWC Group Design Standards. The average effect size 
was 0.03 (SE = 0.05) and was not statistically significant (p = .56). The effect size translates to an improvement 
index of +1. 


To answer the fourth research question—do any of the program components explain differences in 
effectiveness—we attempted to conduct moderator analyses using the intervention’s components. The analysis 
of program variation—the extent to which the programs’ effectiveness varied—indicated that program 
effectiveness was similar. Despite the similarity, and in recognition of the importance of answering the research 
question, the moderator analyses were conducted. Each moderator analysis was conducted within each of the 
two outcome domains, but none of the results were statistically significant. The full results of those analyses are 
provided in tables A-4 and A—5 in the technical appendix. 


Table 1. Summary of findings on distance learning programs from nine studies that Met WWC Group 
Design Standards 


Effectiveness Evidence meeting WWC standards 


Average effect size 
Outcome domain (Gimlaler-lnem-lanelp) NaavelodVcclant=larenlavel=.4 Number of studies | Number of students 
English language arts 0.20 (0.03)* +8 5 6,853 
Mathematics 0.03 (0.05) +1 4 5,393 


* Significant at p < .05. 
Note: Heterogeneity statistics for the ELA analysis are as follows: Q = 7.67, df = 4, p = .10, t? = 0.01. Heterogeneity statistics for the mathematics analysis 
are as follows: Q = 4.27, df = 3, p=.23, tT” =0.01. 


Implications 

The results of the rapid evidence review revealed 15 studies that Met WWC Group Design Standards and three 
studies that met ESSA Tier 1 levels of evidence. Some administrators and education stakeholders may, therefore, 
identify programs from this review that fit their needs. Broadly, however, the implications are clear: There are 
simply not enough high-quality program evaluations, especially randomized controlled trials, conducted in the 
field of distance learning. This is true for all the program types, age groups, and outcomes we examined. More 
evidence is needed overall before any strong conclusions can be drawn regarding the effectiveness of distance 
learning programs. 


The review revealed three effective programs for students in grades K-8 that Met WWC Group Design Standards 
Without Reservations with findings that met ESSA level Tier 1, which education administrators and policymakers 
might consider. The evidence gap map also revealed that the programs that used more innovative approaches— 
programs that were adaptive (i.e., ASSISTments)—were designed for students in grades K-8. Innovative 
approaches such as these may offer the best opportunity for success in remote learning, warranting additional 
research on whether these approaches are effective for older students. 
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Additional rigorous research on distance learning for high school students is especially needed. Only two studies 
were eligible for review, only one of these Met WWC Group Design Standards, and one of the two included 
students in grade 9 only. Given the appropriateness of distance learning technology for high school-aged students, 
an obvious need exists for greater program evaluation across all content areas. 


The implications differ slightly for postsecondary education. Although additional research is indeed needed for 
these students, a review of ineligible studies (see table A-3 in the technical appendix) revealed that most 
postsecondary studies do not measure an eligible outcome. If alignment with the WWC is preferable, 
postsecondary researchers should consider measuring outcomes such as final course grades, college-level credits 
earned, or degree attainment in addition to proximal measures of knowledge attainment that are not eligible for 
WWC review. 


Finally, studies examining the effect of distance learning programs on social-emotional outcomes are lacking. The 
review revealed only one study that examined these outcomes and it did not Meet WWC Group Design Standards. 
Given the nature of the pandemic and its resulting challenges, it is more important than ever to assess social- 
emotional health in students. 


Limitations 


Perhaps the most profound limitation of this rapid evidence review is the fact that, at the time the programs in 
this review were evaluated, a global pandemic was not occurring simultaneously. Education stakeholders, in 
addition to the parents, guardians, teachers, or the students themselves, participated in the program evaluations 
under much different circumstances. Therefore, should an education administrator adopt one of the promising 
distance learning programs, some measure of caution must be considered. The results of the program evaluations 
may not fully translate to the circumstances of the current situation. 


Important evidence may have been excluded because of the necessary constraints of this rapid and streamlined 
approach. The rapid evidence review only included studies that (a) were peer reviewed, (b) were in the ERIC 
database, (c) were RCTs, and (d) reported at least one positive and statistically significant result on a student 
outcome. As a result, rigorous quasi-experimental designs and RCTs with null or negative results and peer- 
reviewed studies that were not indexed in ERIC were not included in the current review. During the screening 
stage, 49 studies were removed because they (a) used quasi-experimental designs, (b) had negative or null results, 
or (c) assessed an outcome not included in the RISP. Broadening the scope of the review to include quasi- 
experimental designs and studies without significant positive findings has the potential to more than double the 
total number of studies reviewed. 


In addition, the review was intentionally restricted to studies testing programs that are implemented entirely at a 
distance; many studies of hybrid programs that include both distance and in-person components were excluded 
during the screening stages. However, given that many students will experience a combination of in-person and 
remote instruction in academic year 2020/21, understanding the efficacy of hybrid approaches could be critical. 


The WWC also set out to explain differences in program effectiveness by conducting moderator analyses. The 
findings indicated that the reviewed programs were similar. As a result of this similarity, the moderator analyses 
did not reveal that some program characteristics provided greater effectiveness than other characteristics. This 
may be due to a lack of statistical power. The lack of significant findings may also be partly attributable to the 
review’s eligibility criteria and the resulting studies available for meta-analysis. Six of the 15 studies used a non- 
business-as-usual comparison group—in which an enhanced or optimized version of a program was compared 
with a base version—and so could not be included in the meta-analysis. Whatever the reason, should future 
syntheses be conducted, moderator analyses should be performed again to test for differences in program 
effectiveness. 
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Conclusions 


Few education leaders and stakeholders could have possibly been prepared for the pivot to distance learning in 
March 2020. The resulting sharp transition to remote education created enormous pressure on administrators to 
provide adequate resources and distance learning programming. Now that education stakeholders have 
committed to these changes, distance learning may be a solution that districts return to in 2021 and beyond. The 
New York City Department of Education, for example, decided that distance learning will be used on snow days 
and other traditional days off such as Election Day (Cramer, 2020). If the need arises in the future, educators may 
pivot back to distance learning to maximize instructional time for students. In other words, regardless of the global 
pandemic’s longevity, distance learning may continue to be used by educators for a variety of reasons. 


The review did reveal several distance learning products that education stakeholders should consider, depending 
on their situation. Equally important, it found that distance learning currently requires much greater investment, 
study, and rigorous evaluation. Through increased resources and a greater understanding of the existing evidence 
on effective distance learning programs, the WWC may be able to decrease the burden placed on education 
practitioners as they grapple with an instructional landscape that is shifting because of the coronavirus pandemic 
and may help them to better support their students now and in the future. 
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